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The purpose of this study was to determine the effectiveness 
of a teacher professional development (PD) program as mea¬ 
sured by the extent that participants have continued to use 
lessons and materials up to three years after the PD experi¬ 
ence. The PD was delivered online and structured by five key 
characteristics of effective PD. A causal-comparative analysis 
of 65 participants of the Food, Land, and People (FLP) PD 
was performed using a sustained implementation scale (SIS). 
Participants completed an online survey answering certain 
demographic variables and indicating the number of lessons 
and activities they had used from the FLP PD. The SIS mod¬ 
el was used to create weighted FLP use scores and compare 
participants within each group. Results suggest participants 
continued to use PD materials up to three years after the PD. 
Further analysis found significance between SIS scores and 
years of teaching experience. Results suggest that the FLP 
PD program was effective in obtaining long- term continued 
use of materials. 
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INTRODUCTION 

Professional development (PD) in education is intended to improve 
the knowledge and skills of teachers to increase student learning (Guskey, 
2000) and has been used to motivate teachers for many years (Saylor & 
Kehrhahn, 2003). When educators introduce new curricula, implement new 
teaching strategies, or incorporate new ideas, some type of PD has often 
been used, which has led to the development of a wide variety of PD mod¬ 
els and methods (Darling-Hammond & Richardson, 2009). 

Traditionally, PD has been conducted face-to-face, allowing participants 
and presenters opportunities to see and interact with each other (Cole & 
Styron, 2006). Currently, advances in technology have made it possible for 
PD to be carried out online with two common types of delivery: synchro¬ 
nous and asynchronous. Synchronous requires participants to log on to a 
computer at specific times in order to participate with others in the course, 
while asynchronous allows participants to log on and work according to 
their own schedule (Brown & Green, 2003). In the past, participants of on¬ 
line courses were not able to see or personally interact with the providers 
of the PD. Instead, the instructor provided feedback to participants through 
email, online discussions, and/or information postings. 

Regardless of the delivery method, literature on teacher PD has identi¬ 
fied five characteristics that are considered key to providing effective PD 
programs (a) a focus on teaching specific content (Loucks-Horsley, Love, 
Stiles, Mundry, & Hewson, 2003); (b) the integration of specific teaching 
practices or pedagogy into the PD (Foulger, 2005); (c) the engagement of 
participants in active learning (Boyle, Lamprianou, & Boyle, 2005); (d) col¬ 
lective participation of teachers from the same grades and/or subject (Snow- 
Renner & Lauer, 2005); (e) delivery with an extended duration (Jeanpierre, 
Oberhauser, & Freeman, 2005). 

Evaluation of teacher PD is essential in determining how effective the 
PD has been, and this evaluation should be conducted on several different 
levels (Guskey, 2000, 2002; Guskey & Yoon, 2009). Guskey explained that 
the levels of evaluation are hierarchal in nature and should also be consid¬ 
ered hierarchal in importance. The first and lowest level of evaluation is the 
measurement of participants’ reactions to the PD session. The second level 
is whether the participants learned something from the PD experience. The 
third level looks at support from the organization that is facilitating the PD. 
The fourth level looks at whether participants continue to use the new in¬ 
formation, skills, or strategies taught during the PD program. The fifth and 
highest level evaluates student learning outcomes as a result of teacher 
change (Guskey, 2000). Although each level of evaluation is important, this 
paper will focus on level four to determine if teachers participating in an on¬ 
line PD program continued to use materials presented in the course. 
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REVIEW OF LITERATURE 


Framework for PD 

Teacher PD has a long history (Bergquist & Phillips, 1975; Cochran- 
Smith & Lytle, 1999; Hargreaves, 2000; Hunt & Michael, 1983) and is of¬ 
ten heralded as one of the most effective ways in which to impact teacher 
practice (Abadal-Haqq, 1995; Almazroa, 2013; Littlejohn, 2002; National 
Center for Education Statistics [NCES; -Nicholls, 2014; 2001; Plecki, 2000; 
Showers, Joyce, & Bennett, 1987; Wilson & Berne, 1999). Although identi¬ 
fied by several labels other than PD, such as teacher or staff development 
or teacher in-service, PD can be defined as those processes and activities 
defined to enhance the professional knowledge, skills, and attitudes of edu¬ 
cators so that they may, in turn, improve student learning (Guskey, 2000). 
Traditionally used formats and delivery methods of teacher PD are conduct¬ 
ed face-to-face, such as workshops, lectures, or college courses that may in¬ 
clude printed or visual materials and/or focus on discussion (Guskey, 2000). 

The increased popularity of the Internet, starting in the mid-1990s, made 
way for a new format: online PD (Cole & Styron, 2006). Despite not hav¬ 
ing personal interaction between instructor and participants, online PD has 
steadily grown in popularity (Bates, Phalen, & Moran, 2016; Hodgson, 
2002; Littlejohn, 2002; Young, Chan, & Lin, 2002). Schools, colleges/uni¬ 
versities, and other PD agencies have begun to offer either full or partial 
online PD. 

Online Delivery 

As society becomes more technologically literate, more people are turn¬ 
ing to the Internet as a source to receive training and further learning. Un¬ 
like traditional PD, asynchronous web-delivered PD allows participants to 
access the course materials when it is best for them and at a comfortable 
pace (Bates, Phalen, & Moran, 2016; Cole & Styron, 2006). This format 
eliminates rigid schedules and time-frames because it is available 24 hours a 
day, seven days a week (Bintrim, 2002). 

Online PD is also easily accessible because participants can log on and 
work from any geographical location. Additionally, web-delivered PD has 
an added incentive because overhead costs (buildings, furniture, and utili¬ 
ties) are reduced or often eliminated (Brown & Green, 2003). 

Key Characteristics of Effective Teacher PD 

Research on the evaluation of teacher PD has identified key charac¬ 
teristics of effective PD models. Guskey, in an early meta-analysis on the 
characteristics of effective PD, synthesized 13 studies from 1995 through 
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2002 that identified key characteristics. His initial review found lists rang¬ 
ing between 6-16 major characteristics, which he synthesized into six key 
characteristics. Since Guskey’s early work in 2002, an additional review 
of the literature combined two of the previous six characteristics into one, 
leaving five major characteristics (Rasmussen, 2008). The most often cited 
characteristic is a focus on teachers’ content and pedagogical knowledge, 
which enhances the participant’s academic content knowledge and provides 
training in specific teaching strategies (Boyle et al., 2005; Foulger, 2005; 
Jeanpierre et al., 2005; National Staff Development Council, 2001). The 
next highest reported characteristic was duration, or providing at least 20+ 
hours of PD (Boyle et al., 2005; Jeanpierre et al., 2005; Meyer & Barufaldi, 
2003). Third on the list were teacher PD programs that promote collegial- 
ity and collaboration where teachers from the same schools, grade levels, 
or subject matter were able to interact with each other (Dearman & Alber, 
2005; Sternberg, 2006). The fourth highest area was having an evalua¬ 
tion component built into the PD (Bredeson, 2003; Elmore, 2002; Guskey, 
2000, 2002; Tallerico, 2005). The fifth area was PD programs that were re¬ 
form based, or required participants to actively engage in either activities 
or pedagogical practice (Snow-Renner & Lauer, 2005; Supovitz & Turner, 
2000). Finally, PD programs that provided modeling of teaching content by 
instructors for participants was also cited as being effective (Birman et al., 
2000). 

Evaluation of PD 

It is essential to evaluate PD as it provides a base to ascertain effective¬ 
ness and needed changes (Bredeson, 2003; Desimone, Smith, & Phillips, 
2013; Elmore, 2002; Guskey, 2000, 2002, 2012; Guskey & Yoon, 2009; 
Marrongelle, Sztajn, & Smith, 2013; Tallerico, 2005). Although consensus 
exists, as revealed in the literature, regarding effective teacher PD practic¬ 
es and methods for evaluating them, the majority of studies reporting what 
constitutes an effective PD activity has only measured at Guskey’s (2000) 
first three levels (i.e., participants’ reactions, learning, and support and 
change) generally through the administration of surveys (Lethwaite, 2005; 
Orrill & Intermath-team, 2006). These lower levels of evaluation are not 
sufficient for detennining the effectiveness of PD programs (Early & Por- 
ritt, 2014). 


PROBLEM STATEMENT 

Although researchers agree that teacher PD should be structured around 
key characteristics to be effective, most evaluations for effectiveness 
of teacher PD are only performed at the lower three levels of evaluation. 
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Systematic evaluation of the two higher levels of Guskey’s evaluation mod¬ 
el (participant continued use and student achievement) is needed. There are 
relatively few hierarchal evaluations using levels four or five that have been 
conducted on teacher PD. Of these few studies, some were poorly conduct¬ 
ed and had problems such as not stating how long after the PD they were 
evaluated or evaluating for continued use during the PD (Guskey, 2002). 

There continues to be a gap in the knowledge of whether PD programs 
that are structured by the key characteristics of effective PD really are effec¬ 
tive in promoting long-term continued use of materials and strategies from 
the PD. It is the goal of the present study to address this gap in the liter¬ 
ature. The researchers used a program known as Food, Land, and People 
(FLP), an asynchronous graduate level online course PD developed for K-6 
educators to increase knowledge about agricultural/environmental literacy 
while meeting statewide mandatory curriculum standards in science, social 
studies, and healthy lifestyles (Utah Agriculture in the Classroom, 2007). As 
part of this PD experience, teachers are taught strategies to improve class¬ 
room practice and are provided with lesson plans and activities to imple¬ 
ment in their classrooms. The course is offered as a Pass/Fail grading sys¬ 
tem where teachers may earn one to three Utah State University semester 
credit(s). FLP is designed as a hybrid program combining some aspects 
of online PD and face-to-face meetings. The FLP PD participants attend a 
face-to-face orientation followed by an asynchronous online independent 
study course. Participants are allowed 12 months to complete the course 
from the time of registering. Although the course has been in place for sev¬ 
eral years, systematic evaluation of effectiveness on the learning of students 
has yet to be determined. 

FLP and Key Characteristics of Effective PD 

As mentioned, studies have shown that five essential characteristics 
have been identified that influence the effectiveness of PD (Birman et ah, 
2000; Bredeson, 2003; Boyle et ah, 2005; Dearman & Alber, 2005; Elmore, 
2002; Foulger, 2005; Guskey, 2000, 2002; Jeanpierre et ah, 2005; Meyer & 
Barufaldi, 2003; National Staff Development Council. 2001; Rasmussen, 
2008; Snow-Renner & Lauer, 2005; Sternberg, 2006; Supovitz & Turner, 
2000; Tallerico, 2005). In developing the FLP course, these five crucial fea¬ 
tures were taken into consideration. Table 1 shows how these characteristics 
were implemented in the course components. 
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Table 1 

Food, Land, and People and Key Characteristics of Effective PD 


Characteristic 

Food, Land, and People components 

Specific content or pedagogy 

Onsite orientation 

Project ideas 

Reflection journals 


Integrated into curriculum 

State aligned less plans in: 

Science 

Social studies 



Healthy lifestyles 

Active learning 

Documented instruction hours 

Curriculum integration 

Collaboration 

Onsite orientation 

Faculty room postings 

Emailing teaching tips 


Extended duration 

Onsite orientation 

3 hours 


Classroom instruction 

28 hours 


Projects 

10 hours 


Final strategy report 

10 hours 


Content and Pedagogy 

Participants of FLP receive specific content training during the onsite 
orientation. Participants receive instruction on pedagogy several ways dur¬ 
ing the course. Some of these ways are as part of the onsite orientation, by 
reflecting and sharing teaching experiences with the instructor and other 
teachers, and by completing a journal form describing implementation strat¬ 
egies and evidence of student learning. 

Integrated into the Curriculum 

FLP requires participants to implement course content into their class¬ 
rooms and curriculum. After completing a lesson or activity, participants 
submit an online journal form that listed the lesson plan title, number of in¬ 
structional hours spent delivering the lesson, strengths of the lesson and/or 
improvement suggestions, additional activities that were used, integration 
strategies or other resources utilized with the lesson, and an explanation of 
the evidence that the students understood the standards/objectives. 
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Active Learning 

Participants practice some lessons on other participants during the onsite 
orientation, then later with their own elementary students. Participants are 
required to complete a minimum number of instruction hours, which varied 
depending on the amount of credit hours the participant was taking. Upon 
completion of each lesson, participants complete a teaching journal where 
they reflect on their experience teaching, describing the strengths and weak¬ 
nesses of their lesson and its implementation. 

Collaboration 

Participants have two major sources of collaboration. First, participants 
collaborate about specific content knowledge in agricultural-related science, 
social studies, and healthy lifestyles during the onsite orientation. Second, 
participants collaborate through the course website’s Faculty Room, an 
electronic bulletin board where participants share ideas or tips about teach¬ 
ing using the FLP curriculum a minimum of four times during the course. 
In addition to posting in the Faculty Room, participants are required to 
email the instructor and share their ideas and teaching tips. 

Extended duration. Many studies suggest that PD needs to provide par¬ 
ticipants at least 40 contact hours to be effective (Boyle et ah, 2005; Jean- 
pierre et ah, 2005; Meyer & Barufaldi, 2003). Participants who take FLP 
exceed the 40-hour minimum. Table 2 shows the number of contact hours 
FLP participants received. 


Table 2 

Food, Land, and People Contact Hours 


Activity 

Estimated contact hours 

Onsite orientation 

3 

Projects 

10 

Classroom instruction 

28 

Final strategy report 

10 

Total hours 

51 


PURPOSE AND RESEARCH QUESTIONS 

The primary purpose of this study is to determine the extent to which par¬ 
ticipants continue to use the lesson plans and activities obtained from FLP. 
This hybrid PD program meets all five of the key characteristics of effective 
PD as defined in the literature (Guskey, 2003; Loucks-Horsley et ah, 2003; 
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Rasmussen, 2008; Snow-Renner & Lauer, 2005). Evaluations previously 
conducted on the lower two levels have been very positive, however, it is im¬ 
portant to evaluate FLP at level four (participant continued use) to determine 
the extent that participants have continued to use the lessons and strategies 
taught during the PD program. This evaluation was performed by creating 
a Sustained Implementation Scale (SIS) model that gives each participant a 
weighted score correlated to the amount of lessons and activities used since 
the PD experience and the amount of time since taking FLP PD. The follow¬ 
ing research questions guided this study: 

• With what frequency do participants continue to use the Food, Land, 
and People curriculum? 

• What variables explain the variance in SIS? 


PROCEDURES 


Population and Sample 

The participants from this study came from a census group consisting of 
172 elementary teachers from across the state of Utah and represented 21 of 
the 40 school districts in the state. All participants were selected because of 
their enrollment in Utah State University’s FLP online course over a three 
year period. Of the participants, 94% were female and 6% male. In total, 
participants had an average of 16.18 years (SD= 8.22) teaching experience 
with 63% holding bachelor’s degrees and 37% master’s degrees. Twenty- 
six percent taught PreK-K through 2nd grade, 53.9% taught 3rd through 6th 
grade, and 20.1% taught multiple grades. Each participant freely chose and 
paid to attend the course and did not receive any sort of stipend or monetary 
reward from the facilitators of the PD program. 

Research Design 

This study used a causal-comparative research design, also known as 
an ex post facto design. This type of design is used to discover the possi¬ 
ble causes and effects of a behavior pattern or personal characteristics by 
comparing individuals displaying the particular behavior pattern with indi¬ 
viduals who do not display the behavior pattern (Borg & Gall, 1989). The 
causal-comparative method is regularly used in education studies when ex¬ 
perimentation is not possible. It identifies a cause for a particular behavior 
after some treatment has been administered. In this study, the behavior we 
were investigating was continued use of PD lessons and activities and the 
possible reasons participants continued to use the provided PD materials. 
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Data and Instrumentation 

The researcher developed an online survey instrument that was used to 
identify which teacher participants continued to implement the FLP curric¬ 
ulum and which demographic variables contributed to their continued use. 
The survey instrument was administered one time via email to each partici¬ 
pant of the FLP program who had completed the PD during a time period 
of one to three years prior to the study. In addition to the initial email, four 
follow up reminder emails were sent. This survey collected participant data 
on nine selected personal and professional characteristics: (1) gender, (2) 
number of years teaching experience, (3) highest degree earned, (4) type of 
teaching license held, (5) grade level primarily taught, (6) location of the 
school (rural or urban), (7) percentage of low SES students at the school, (8) 
percentage of non-white students at the school, and (9) whether the school 
met NCLB requirements of Annual Yearly Progress (AYP) the year the 
teacher enrolled in FLP. Further, the survey asked participants to indicate 
all lesson plans or activities they had used since taking FLP. The survey had 
one column of lessons and activities for participants who had taken FLP 
one year prior, two columns for those who took it two years prior, and three 
columns for those who took FLP three years prior. There was also a column 
for the grade taught. For example, participants who indicated they taught 
pre-kindergarten only had lesson plans and activities that were available 
to that grade. Participants who taught multiple grades indicated for which 
grade each lesson or activity was used. 

A panel of five experts from elementary science education and agricul¬ 
tural education analyzed the survey questions for validity and appropriate¬ 
ness. Internal consistency for the scale items was measured using a split half 
of the instrument (Borg & Gall, 1989). Further, a post-hoc reliability analy¬ 
sis of the survey instrument was performed to determine if the instrument 
had an acceptable measure of reliability. One limitation of the instrument 
was that it contained no variables appropriate for a Cronbach’s coefficient 
alpha. Therefore, the panel determined to calculate an internal consistency 
measure of reliability. Borg and Gall (1989) stated, “internal consistency 
can be determined from a single administration of a single form of the test” 

(p. 260). 

A sustained implementation scale (SIS) was used to classify the amount 
of continued use of FLP curriculum a participant has reported. This mea¬ 
surement was created by determining the number of PD provided lessons 
and activities a participant used each year and then assigning weight to 
the scores (Linn & Haug, 2002). The SIS model allows for the raw survey 
numbers to be balanced for more accurate analysis. The reported number of 
lessons or activities used in the third year following the PD program was 


154 


Rasmussen and Byrd 


weighted at three times the reported number of lessons or activities used, 
because continued use three years after the PD program signifies a stronger 
indication of impact than someone who uses lessons immediately follow¬ 
ing the PD. The number of lessons or activities used in the second year fol¬ 
lowing the PD was weighted at two times the reported number used. The 
number of lessons and activities used in the first year following the PD was 
weighted by one times the number reported. A total SIS was determined for 
each participant by summing each of the weighted scores across years 1, 2, 
and 3. 

Limitations 

One limitation of this study is the use of self-reported data. Baird and 
Ozler (2012) suggest that people often over report their work (especially 
when it may appear to be for the greater good) than what they have actu¬ 
ally done. This study relied on self-reported data because of the geographic 
spread and high number of participants using the PD materials. Despite the 
inflated number of PD lessons being used, analysis of SIS across years re¬ 
mains valid. It is reasonable to suppose that participants consistently over¬ 
report use from year to year (2012), thus eliminating any skew of the SIS 
data. 

Another limitation of this study is age of the study. Three year’s worth 
of data (2005-2008) were collected during this study. During the time since 
data were collected, the components of effective PD (Bayar, 2014) and the 
recommended levels of evaluation of PD (Soebari & Aldridge, 2015) have 
remained consistent, thus making analysis of the data important. 

RESULTS 

PD programs that do not result in some form of participant change can¬ 
not be considered effective (Johson, Bolshakova, & Waldron, 2016; Show¬ 
ers et al., 1987). This premise was a foundational issue in this study. The 
first research question examined with what frequency do participants con¬ 
tinue to use the FLP curriculum. Participants in each group of this study 
were asked to report which FLP lessons and activities they had used each 
year since taking the FLP program as an indication of participant change. 
These totals were entered into the SIS evaluation model. This model used 
a system of weighting scores to provide a systematic means of comparing 
participants (i.e., the combined number of lessons and activities used during 
the first year after the PD experience are weighted one time, those after the 
second year are weighted by two and after the third year were weighted by 
three). A final total SIS was calculated by summing each SIS score for each 
participant. 
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The first year following the PD program is considered the implementa¬ 
tion year. Groups 1 and 2 had very close SIS scores as is seen in Table 3. 
Although Group 3’s SIS appears slightly higher than the other two groups, 
because of the standard deviations ranging from 11.72 to 13.14 points, the 
scores are actually quite similar. Year 1 SIS scores indicate the average 
number of lesson plans and activities used by participants the first year after 
talcing FLP. 


Table 3 

Sustained Implementation Scales 



Group 1 


Group 2 


Group 3 

Year 

M 

SD 

M 

SD 

M 

SD 

1 

12.78 

12.96 

11.92 

11.72 

17.30 

13.14 

2 



20.83 

19.80 

20.00 

17.36 

3 





25.57 

23.01 

Total 

12.78 

12.96 

32.75 

31.31 

62.87 

41.80 


The second year after the PD program is the beginning of a continued 
use stage as defined by the SIS model. As shown in Table 3, the Year 2 SIS 
is very similar for each Group. The Year 2 SIS is almost two times greater 
than Year 1 scores. While Year 2 participant numbers are larger than Year 
l’s participants, it does not mean the participants have used more lesson 
plans and activities the second year. In actuality, the average number of ma¬ 
terials used has slightly decreased. While Group 2 had reported using al¬ 
most 12 lessons or activities in Year 1, in Year 2 they reported using only 10 
lessons and activities. While Group 3 had reported using nearly 18 lessons 
and activities during Year 1, they also reported using only 10 lessons or ac¬ 
tivities during Year 2. The reason the SIS scores are larger in Year 2 is be¬ 
cause for every lesson plan or activity reportedly used, the number is multi¬ 
plied by two. It is felt that use of materials two years after a PD experience 
is a strong indication of the impact the PD program had on the participants’ 
teaching and is thus weighted two times greater than the amount of use the 
first year after the program. 

The use of materials the third year following the PD experience indi¬ 
cates a strong continued use correlation. Group 3 was the only group with 
a Year 3 SIS score because they are the only ones that had completed FLP 
3 years previous. As seen in Table 3, the average SIS scores in Year 3 in¬ 
creased by 5 points. This result does not indicate a significant increase in 
score. It shows that, on average, participants were only using eight lesson 
plans or activities the third year after the PD experience. Year 3 SIS scores 
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were created by multiplying the number of reported lesson plans or activi¬ 
ties being used by three. Participants who are still using materials from the 
PD activity three years after the program have indicated that the PD pro¬ 
gram has had a significant impact on their use of the PD provided lesson 
plans and activities, and thus it receives a weighting of 3. 

Table 3 also shows the total SIS scores for each group. Group 3 had the 
highest total SIS, Group 2 had the second highest, and Group 1 the low¬ 
est. This is to be expected because total SIS scores are calculated by sum¬ 
ming Year 1 SIS, Year 2 SIS, and Year 3 SIS scores. Groups 1 and 2 do not 
have as many SIS scores to put into the equation as Group 3 does. Likewise, 
Group 1 does not have as many SIS scores as Group 2. We cannot compare 
total SIS scores across groups. Group 1 scores can only be compared within 
Group 1, Group 2 scores can only be compared within Group 2, and Group 
3 can only be compared within Group 3. 

SIS scores reveal a declining trend in the number of lesson plans and ac¬ 
tivities used each subsequent year following the FLP experience. Although 
the number of FLP lessons and materials participants use has slightly de¬ 
creased, the average SIS scores have increased. These results offer valid¬ 
ity to the research on effective PD programs by confirming that effective 
PD programs result in changes in teacher attitudes or practice (Boyle et ah, 
2004, 2005). The rate that participants have discontinued use was much 
smaller than the weighting of the scores in subsequent years. This is an indi¬ 
cator that the FLP PD was effective in promoting participant continued use. 

The second part of the study examined which variables explained vari¬ 
ance in SIS. A multiple linear regression was performed on the responses 
from the participants who took FLP during the 2005-06 academic school 
year (Group 1). In the regression model, the total SIS was used as the de¬ 
pendent variable with the other selected demographic variables chosen as 
the independent variables (see Table 4). Overall, the regression model indi¬ 
cated no statistically significant relationship among the combined selected 
demographic variables and total SIS, F( 8, 6) = 3.368, p = .078. However, 
the R = .904. Therefore, 81.8% of the variance in SIS was explained by the 
eight demographic variables. For this analysis, the researcher entered nine 
variables into the model. No variance was measured in “AYP” and the vari¬ 
able was deleted by the statistical program. 
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Table 4 

Multiple Linear Regression for Group 1 Participants 


Variable 

Beta 

t 

P 

Gender 

0.308 

1.371 

0.116 

Years teaching 

0.895 

3.876 

0.008* 

Highest degree 

-0.667 

-2.418 

0.052 

Type of license 

0.550 

2.691 

0.036* 

Grade level 

0.457 

1.769 

0.127 

School location 

-0.171 

-0.622 

0.557 

SES 

0.376 

1.272 

0.251 

Percent ethnic 

-.661 

-2.392 

0.054 

AYP 

+ 

+ 

+ 


* Indicates significance at the .05 level. 

+ Indicates that the model deleted the variable from the analysis due to missing correlations. 

In the regression analysis, two variables indicated a statistical signifi¬ 
cance in the model. There was a statistically significant relationship be¬ 
tween total SIS and “Years Teaching,” p = .008 as well as between total 
SIS and “Type of License,” p = .036. Further analysis indicated that as a 
teacher had more years of teaching experience and a higher level of teach¬ 
ing license, the total SIS tended to be higher. Years teaching and the type of 
teaching license were both factors that influenced whether participants con¬ 
tinued to use the FLP materials. 


Table 5 

Multiple Linear Regression for Group 2 Participants 


Variable 

Beta 

t 

P 

Gender 

0.134 

0.400 

0.695 

Years teaching 

-0.251 

-0.719 

0.485 

Highest degree 

-0.104 

-0.358 

0.726 

Type of license 

0.074 

0.204 

0.842 

Grade level 

0.304 

0.966 

0.352 

School location 

-0.083 

-0.174 

0.865 

SES 

0.037 

0.081 

0.936 

Percent ethnic 

-0.276 

-0.503 

0.624 

AYP 

0.459 

1.322 

0.209 
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A multiple linear regression was performed on the responses from 
the participants who took FLP during the 2004-05 academic school year 
(Group 2). In the regression model, the total SIS was used as the depen¬ 
dent variable with the other selected demographic variables chosen as the 
independent variables (see Table 5). Overall, the regression model indicated 
no statistically significant relationship among the combined selected de¬ 
mographic variables and total SIS, F(9, 13) = .535, p = .825. However, the 
R = .520. Therefore, 27.0 % of the variance in SIS was explained by the 
nine demographic variables. For this analysis, the researcher entered nine 
variables into the model. 


Table 6 

Multiple Linear Regression for Group 3 Participants 


Variable 

Beta 

t 

P 

Gender 

-0.127 

-0.534 

0.603 

Years teaching 

0.578 

2.619 

0.022* 

Highest degree 

0.088 

0.367 

0.720 

Type of license 

0.087 

0.372 

0.717 

Grade level 

0.230 

0.937 

0.367 

School location 

0.273 

0.962 

0.355 

SES 

-0.699 

-1.381 

0.193 

Percent ethnic 

1.007 

1.881 

0.084 

AYP 

0.305 

1.362 

0.198 


* Indicates significance at the .05 level. 

Responses of participants who took FLP during the 2003-04 academic 
year (Group 3) were analyzed using a multiple linear regression. SIS was 
used as the dependent variable with selected demographic variables being 
used as the independent variables (see Table 6). The regression model indi¬ 
cated no statistically significant relationship among the combined selected 
demographic variables and total SIS, F( 9, 12) = 1.618, p = .215 with R = 
.740. Over half (54.8%) of the variance in SIS was explained by the nine 
demographic variables. Nine variables were used for this analysis. 

In the regression analysis, one variable indicated a statistical significance 
in the model. There was a statistically significant relationship between to¬ 
tal SIS and “Years Teaching,” p = .022. Further analysis indicated that as 
a teacher had more years of teaching experience, the total SIS tended to be 
higher. The number of years teaching experience was a significant factor in¬ 
fluencing Groups 1 and 3 on their continued use of FLP materials after the 
PD program. 
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DISCUSSION 

SIS scores reveal a declining trend in the number of lesson plans and ac¬ 
tivities used each subsequent year following the FLP experience. Although 
the number of FLP lessons and materials participants use has slightly de¬ 
creased, the average SIS scores have increased. These results offer validity 
to the research on effective PD programs by confirming that effective PD 
programs, online or otherwise, result in changes in teacher attitudes or prac¬ 
tice (Boyle et al., 2004, 2005). The rate that participants have discontinued 
use was much smaller than the weighting of the scores in subsequent years. 
This is an indicator that the FLP PD was effective in promoting participant 
continued use of materials for teaching lessons. 

Historically, research indicates that teachers have been resistant to 
change in their pedagogy (Cuban, 1988; Fullman, 1991). Yet more recent 
research (Richardson, 2001; Sharma, 2016) has indicated that teachers 
will make changes in their teaching practices when they are involved in 
PD activities structured by the five key characteristics of effective PD list¬ 
ed above. While one of the main focuses of this PD program was to have 
teachers use FLP lesson plans and activities, it was also structured by key 
characteristics of effective PD as discussed in the literature review. 

Selected Personal and Professional Characteristics 

The review of literature on teacher PD programs indicates that several 
demographic variables are often indicators of continued use (Schonlau, 
Fricker, & Elliott, 2002). Each of the demographic variables was analyzed 
in a multiple regression using total SIS scores as independent variables. The 
SIS scores in each group were statistically similar and indicate that partici¬ 
pants from each group had used between 8-12 lessons or activities each year 
following the PD program. Having a low number of significant variables 
is a sanguine phenomenon because fewer significant items indicate that 
the program is working for a wide demographical population. In Group 1, 
there were two demographic variables that were statistically significant. The 
number of years teaching experience and the type of license the teacher held 
were both significant variables affecting continued use. In Group 2 none of 
the demographic items were found to be significant. In Group 3, the only 
statistically significant variable was the number of years teaching. While 
it is possible that the other demographic variables were factors on whether 
participants continued to use FLP materials, they were not statistically sig¬ 
nificant. 

Teaching Experience 

The more experienced teachers in this study were more likely to con¬ 
tinue using materials after the PD program. One reason is that experienced 
teachers have increased levels of confidence in their teaching abilities, and 
thus are more likely to try new lesson plans and activities (Appleton, 1999). 
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Confidence in teaching abilities normally increases with experience. An¬ 
other possibility is that experienced teachers are looking for new ways to 
energize their teaching and will use new lesson plans and activities to fa¬ 
cilitate this desire (Remillard, 1999). Remillard explained that experienced 
teachers often consider their students’ needs when selecting the curriculum 
to be used in the classroom. This is in contrast to new teachers who focus on 
their own teaching preferences when choosing curriculum. Finally, the more 
experienced teachers readily recognize lesson plans and activities that can 
add to the students learning experiences (Appleton & Kindt, 2002). Expe¬ 
rienced teachers could be considered more experienced consumers of edu¬ 
cational innovations and curricular materials that will be effective with stu¬ 
dents. Teacher’s selection of teaching curriculum is often decided by factors 
such as what the school wants and on existing classroom practice. Further, 
Appleton and Kindt (2002) explain that more experienced teachers more of¬ 
ten choose curriculum based on the needs of their students. 

Teaching License 

The type of teaching license was found to be significant in Group 1. Ac¬ 
cording to the statistical analysis, teachers who held Utah Level 2 or Utah 
Level 3 licenses were statistically more likely to continue using items from 
the FLP PD program. One possible reason is that Utah Levels 2 and 3 li¬ 
censes are only given to teachers who have taught for three or more years. 
As discussed earlier, experienced teachers more often use new curriculum 
materials while inexperienced teachers are more reluctant to try new curric- 
ulums. A second possibility for the significance found is that teachers who 
have earned Levels 2 or 3 licenses are required to have completed 100 hours 
of PD training. Individuals who have experienced a great deal of PD may 
feel more comfortable working with new materials (Appleton, 1999), or are 
able to recognize lesson plans that will help increase learning in their stu¬ 
dents (Appleton & Kindt, 2002). 

Gender 

While gender was not a significant variable in this study, it may have 
been because there were only four male participants. This phenomenon is 
typical of elementary schools. Females make up the greater population of 
elementary school teachers (Cunningham & Watson, 2002; Mills, Martino, 
& Lingard, 2004). Flad this study been performed with middle school, ju¬ 
nior high, or high school teachers where the distribution of male/female 
teachers is much closely balanced, the results may have been different. 
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Annual Yearly Progress (AYP) 

Another variable that was not significant in this study but could be in 
other studies was whether the school had met AYP the year the participant 
took the FLP program. In this study five participant’s schools had not met 
AYP and four participants taught at private schools that were not subject to 
AYP. The disparity between the number of schools meeting and not meet¬ 
ing AYP greatly influenced significance. In fact, the regression model was 
not able to count AYP in the analysis for Group 1 because there were not 
any participant’s schools that had not met AYP. In the past, schools that re¬ 
peatedly did not meet AYP were subject to state imposed sanctions such as 
requirements to provide student tutoring, student’s choice of which school 
they attend, or even school administration takeover by the State Department 
of Education. 

AYP was primarily determined through the use of student standardized 
testing (Cochran-Smith, 2005). Thus schools that had not met AYP were 
prone to put extra pressure on teachers to ensure not only that their students 
pass the tests, but that they also make substantial improvement in their test 
scores. Many districts and schools dictated the curriculum that was to be 
taught and teachers were no longer allowed to choose their own curricular 
materials (Cobb, 2005). Cobb further explained that many schools that did 
not make AYP were now dictating and requiring PD in specific content ar¬ 
eas. This level of management did not allow teachers the freedom to choose 
and teach the curriculum of their choice. 

Recently, AYP has been replaced by a new federal initiative, Every Stu¬ 
dent Succeeds Act (ESSA) (U.S. Department of Education, 2016). This act 
is still in the implementation phase and each state is invited to submit how 
they will specifically design and implement state developed instruction and 
outcomes for all students, it is anticipated that ESSA will provide somewhat 
of a relief from the pressures of AYP as each state will now have more con¬ 
trol of the instruction and assessments happening within each state (Fran- 
quiz, & Ortiz, 2016). 

Percent of Low SES Students 

The percent of low SES students at the schools of the participants were 
not statistically significant variables in whether participants continued to 
use FLP materials. Teachers who teach at schools with high numbers of 
low SES students often experience a different curriculum than teachers at 
schools with higher SES students (Duke, 2000). Duke found major differ¬ 
ences between low SES schools and high SES schools in the areas of the 
amount of printed materials such as books and magazines and the quality of 
the material. Low SES schools did not have as many printed items such as 
books and magazines, and the quality of the printed materials was of much 
lower quality than at schools with higher SES. 
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Percent of Non-white Students 

The percent of non-White students was not a statistically significant vari¬ 
able influencing participant continued use. In a study by Lubienski (2002), 
disparity was found between the number and types of courses being offered 
at schools with high numbers of non-White students. In addition to the lack 
of higher level courses, teachers at non-White schools spent less time teach¬ 
ing reasoning skills and relied heavier upon multiple choice testing. Teach¬ 
ing at schools with high numbers of non-White students could be a factor 
influencing whether teachers use PD materials in their classrooms. 

Size of Community 

The size of the community where the participants teach was not a statisti¬ 
cally significant variable in whether participants continued to use FLP ma¬ 
terials in this study. The variable was investigated because the literature in¬ 
dicated that the size of the community is another variable that lets us know 
school. Larger communities often have a larger financial base than smaller 
schools (Lee, Smerdon, Alfeld-Liro, & Brown, 2000). The amount of rev¬ 
enue at a school often determines the amount of materials and additional 
supplies a teacher has access to. Often teachers working in smaller commu¬ 
nities are dependent upon their own resourcefulness to acquire and use extra 
teaching materials. In this study, size of community was not a variable of 
significance. 

Grade Level 

The grade level the participants teach was not a statistically significant 
variable in whether participants continued to use FLP materials. While not 
significant in this study, the grade level a teacher teaches can influence the 
experiences of that teacher. Teachers at each grade level experience students 
differently as the students are at differing developmental levels (Geary & 
Bjorklund, 2000). In looking at the FLP curriculum, a question of interest 
was whether there were differences in continued use by grade level. If this 
had been the case, a closer examination of the curriculum (lesson plans and 
activities) would have been warranted. However, statistical analysis indicat¬ 
ed no differences. 

Highest degree 

The highest degree obtained by the participants in this study was not sta¬ 
tistically significant in whether participants continued to use FLP materials. 
The FLP PD program taught specific content and then promoted using pre¬ 
made lesson plans to teach this content. It is possible that the highest de¬ 
gree obtained was not significant because the participants who had earned 
advanced degrees had received specialized training in their content areas 
through their degree programs and thus did not believe that the PD content 
lessons would provide their students with any advantage over the current 
curriculum. If this variable had been significant, additional investigation 
into the variable could be warranted. 
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CONCLUSION 

This study suggests that online PD can have a positive impact on influ¬ 
encing participants past their enrollment. In this study the FLP PD program 
can be deemed successful in promoting continued use of materials as de¬ 
termined by the SIS scores of each group of participants. Even though the 
participants did not meet face-to-face with a PD instructor, the material cov¬ 
ered and the lesson plans and activities were used well past the end of the 
course of instruction (Cole & Styron, 2006). By keeping in mind the five 
key characteristics of PD instruction while developing online PD courses 
(Guskey, 2000), PD instructors can readily expect to have greater numbers 
of participants continuing to use the content taught during the PD for longer 
periods of time following the PD. 

The amount of experience of the participants in an online PD course 
can also be a significant factor to consider when developing such courses. 
It is not surprising that more experienced teachers tended to use PD ma¬ 
terials longer, as they most likely are more familiar with their curriculum 
and, therefore, may be more ready to implement new materials (Appleton 
& Kindt, 2002; Rasmussen, 2008). Developers of PD courses may need to 
provide more scaffolding for younger teachers in order to show them how 
new materials can be inserted into their classes. 

Finally, the present study provided an evaluation model for measur¬ 
ing the effectiveness of an online PD program that provides teachers with 
specific lesson plans and activities that they can implement directly in their 
classrooms. This model could be used to measure continued use of materi¬ 
als as is the case in this study, but could also be used to quantify the level of 
continued use of a variety of pedagogical strategies or philosophies. The use 
of the SIS model in evaluating the effectiveness of PD programs in relation 
to participants continued use is valuable. 

Future research in this area should look at continued use of PD materials 
past three years. Comparison of current SIS scores and future SIS scores 
woidd let us know if participants have continued to use the FLP materials. 
Additionally, an analysis of student learning would be a natural follow up to 
a study similar to this one. It would be important to see if continued use of 
PD materials results in greater student achievement. 
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